{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "41f17569-20e2-402d-ae4e-a7a8d55295de",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Qdrant & Audio Data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "903f83d2",
   "metadata": {},
   "source": [
    "![main](../images/main_pic.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "23074526",
   "metadata": {},
   "source": [
    "Welcome to this tutorial on vector databases and music recommendation systems using Python and Qdrant. Here, \n",
    "we will learn about how to get started with audio data, embeddings and vector databases.\n",
    "\n",
    "By the end of this tutorial, you will have a good understanding of how to use vector databases and Python to \n",
    "create your own music recommendation engine."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c45dab1c",
   "metadata": {},
   "source": [
    "## 1. Overview"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6d98a101",
   "metadata": {},
   "source": [
    "The dataset we will be using is called, \n",
    "[Ludwig Music Dataset (Moods and Subgenres)](https://www.kaggle.com/datasets/jorgeruizdev/ludwig-music-dataset-moods-and-subgenres) \n",
    "and it can be found on Kaggle. It was collected for the purpose of music information retrieval (MIR) by \n",
    "[Discogs](discogs.com) and [AcousticBrainZ](acousticbrainz.org), and it contains over 10,000 songs of \n",
    "different genres and subgenres. Bare in mind that the full dataset is 12GB in size so we recommend that \n",
    "you download your favorite genre from the `mp3` directory, and the `labels.json` file. That will be more \n",
    "than enough to follow along for the rest of the tutorial.\n",
    "\n",
    "Once you download the full dataset, you should see the following directories and files.\n",
    "\n",
    "```sh\n",
    "../data/ludwig_music_data\n",
    "├── labels.json\n",
    "├── mfccs\n",
    "│   ├── blues\n",
    "│   ├── ...\n",
    "│   └── rock\n",
    "├── mp3\n",
    "│   ├── blues\n",
    "│   ├── ...\n",
    "│   └── rock\n",
    "├── spectogram\n",
    "│   └── spectogram\n",
    "└── subgeneres.json\n",
    "```\n",
    "\n",
    "The `labels.json` contain all the metadata (e.g. artist, subgenre, album, etc.) associated with each song.\n",
    "\n",
    "The `Spectograms` directory contains spectograms, which are visual representation of the frequencies present \n",
    "in an audio signal over time. It is a 2D graph where the x-axis represents time and the y-axis represents \n",
    "frequency. The intensity of the color or brightness of the graph indicates the strength or amplitude of the \n",
    "frequencies at a particular time. Here is an example of a Spectogram.\n",
    "\n",
    "<p align=\"center\">\n",
    "  <img src=\"../images/mel_specto.png\" />\n",
    "</p>\n",
    "\n",
    "If you've ever wonder what audio data looks like visually, this is one way to visualize it.\n",
    "\n",
    "Let's get our environment set up before we prepare the data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9b7dab81",
   "metadata": {},
   "source": [
    "## 2. Set Up"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c4fa151f",
   "metadata": {},
   "source": [
    "Before you run any line of code, please make sure you have \n",
    "1. downloaded the data\n",
    "2. created a virtual environment (if not in Google Colab)\n",
    "3. installed the packages below\n",
    "4. started a container with Qdrant\n",
    "\n",
    "```bash\n",
    "# with conda or mamba if you have it installed\n",
    "mamba env create -n my_env python=3.10\n",
    "mamba activate my_env\n",
    "\n",
    "# or with virtualenv\n",
    "python -m venv venv\n",
    "source venv/bin/activate\n",
    "\n",
    "# install packages\n",
    "pip install qdrant-client transformers datasets pandas numpy torch librosa tensorflow openl3 panns-inference pedalboard streamlit\n",
    "```\n",
    "\n",
    "The open source version of Qdrant is available as a docker image and it can be pulled and run from \n",
    "any machine with docker installed. If you don't have Docker installed in your PC you can follow the \n",
    "instructions available in the official documentation [here](https://docs.docker.com/get-docker/). After \n",
    "that, open your terminal and start by downloading the image with the following command.\n",
    "\n",
    "```sh\n",
    "docker pull qdrant/qdrant\n",
    "```\n",
    "\n",
    "Next, initialize Qdrant with the following command and you should be good to go.\n",
    "\n",
    "```sh\n",
    "docker run -p 6333:6333 \\\n",
    "    -v $(pwd)/qdrant_storage:/qdrant/storage \\\n",
    "    qdrant/qdrant\n",
    "```\n",
    "\n",
    "Verify that you are ready to go by importing the following libraries and connecting to Qdrant via its Python client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f309ad35",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-06-07 00:35:16.123186: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-06-07 00:35:17.342633: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoFeatureExtractor, AutoModel\n",
    "from IPython.display import Audio as player\n",
    "from datasets import load_dataset, Audio\n",
    "from panns_inference import AudioTagging\n",
    "from qdrant_client import QdrantClient\n",
    "from qdrant_client.http import models\n",
    "from os.path import join\n",
    "from glob import glob\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import librosa\n",
    "import openl3\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc9a31fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = QdrantClient(host=\"localhost\", port=6333)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "87d7fdbc",
   "metadata": {},
   "source": [
    "We will also go ahead and create the collection we will be working with in this tutorial. The dimensions will \n",
    "be of size 2048 and we'll set the distance metric to cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7696ded7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_collection = \"music_collection\"\n",
    "client.recreate_collection(\n",
    "    collection_name=my_collection,\n",
    "    vectors_config=models.VectorParams(size=2048, distance=models.Distance.COSINE)\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "197a01f7",
   "metadata": {},
   "source": [
    "## 3. Data Prep"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9df54f17",
   "metadata": {},
   "source": [
    "We will be using Huggin Face's [datasets](https://huggingface.co/docs/datasets/index) library to read in our data and massage it a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3b9a665",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../data/ludwig_music_data'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = join(\"..\", \"data\", \"ludwig_music_data\")\n",
    "data_path"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9aa859f1",
   "metadata": {},
   "source": [
    "Feel free to change the genre to the one you like the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2e2d3a74",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b9b78ff68e148ee813634220089ea91",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Resolving data files:   0%|          | 0/979 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset audiofolder (/home/ramonperez/.cache/huggingface/datasets/audiofolder/default-0ee03b3edf9a7673/0.0.0/6cbdd16f8688354c63b4e2a36e1585d05de285023ee6443ffd71c4182055c0fc)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['audio'],\n",
       "    num_rows: 979\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_data = load_dataset(\n",
    "    \"audiofolder\", data_dir=join(data_path, \"mp3\", \"latin\"), split=\"train\", drop_labels=True\n",
    ")\n",
    "music_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ff4e5dcd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'audio': {'path': '/home/ramonperez/Tresors/qdrant_org/content/examples/data/ludwig_music_data/mp3/latin/0rXvhxGisD2djBmNkrv5Gt.mp3',\n",
       "  'array': array([ 0.00000000e+00,  1.24776700e-09, -4.54397187e-10, ...,\n",
       "         -7.98814446e-02, -8.84955898e-02, -1.05223551e-01]),\n",
       "  'sampling_rate': 44100}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_data[115]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4f523f46",
   "metadata": {},
   "source": [
    "As you can see, we got back json objects with an array representing our songs, the path to where each \n",
    "one of them is located in our PC, and the sampling rate for each. Let's play the song at index 115 and \n",
    "see what it sounds like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94ac7dc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "player(music_data[115]['audio']['array'], rate=44100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ebfab94b",
   "metadata": {},
   "source": [
    "<iframe style=\"border-radius:12px\" src=\"https://open.spotify.com/embed/track/0rXvhxGisD2djBmNkrv5Gt?utm_source=generator\" width=\"100%\" height=\"152\" frameBorder=\"0\" allowfullscreen=\"\" allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\"></iframe>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5350f8a7",
   "metadata": {},
   "source": [
    "We'll need to extract the name of each mp3 file as this is the unique identifier we'll use in order to \n",
    "get the corresponding metadata for each song. While we are at it, we will also create a range of numbers \n",
    "and add it as the index to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6e68bf6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0010BnyFuw94XFautS2uJp',\n",
       " '00RhgYVH6DrHl0SuZWDp8W',\n",
       " '01k69xxIQGL94F8IfIkI5l',\n",
       " '02GUIyXZ9RNusgUocEQIzN']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ids = [\n",
    "    (\n",
    "     music_data[i] # for every sample\n",
    "     ['audio'] # in this directory\n",
    "     ['path'] # extract the path\n",
    "     .split(\"/\") # split it by /\n",
    "     [-1] # take only the last piece \"id.mp3\"\n",
    "     .replace(\".mp3\", '') # and replace the .mp3 with nothing\n",
    "    ) \n",
    "    for i in range(len(music_data))\n",
    "]\n",
    "index = [num for num in range(len(music_data))]\n",
    "ids[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "907b6195",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'audio': {'path': '/home/ramonperez/Tresors/qdrant_org/content/examples/data/ludwig_music_data/mp3/latin/7yX4WgUfoPpMKZHgqpaZ0x.mp3',\n",
       "  'array': array([ 0.00000000e+00, -1.40022882e-09, -4.44221415e-09, ...,\n",
       "         -9.52053051e-02, -8.90597273e-02, -8.10846481e-02]),\n",
       "  'sampling_rate': 44100},\n",
       " 'index': 978,\n",
       " 'ids': '7yX4WgUfoPpMKZHgqpaZ0x'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_data = music_data.add_column(\"index\", index)\n",
    "music_data = music_data.add_column(\"ids\", ids)\n",
    "music_data[-1]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3724fd24",
   "metadata": {},
   "source": [
    "The metadata we will use for our payload lives in the `labes.json` file, so let's extract it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "35bc6668",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tracks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>000QWvZpHrBIVrW4dGbaVI</th>\n",
       "      <td>{'otherSubgenres': {'L': [{'S': 'electronic---...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0010BnyFuw94XFautS2uJp</th>\n",
       "      <td>{'otherSubgenres': {'L': [{'S': ' world'}, {'S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0055LRFB7zfdCXDGodyIz3</th>\n",
       "      <td>{'otherSubgenres': {'L': []}, 'artist': {'S': ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>005Dlt8Xaz3DkaXiRJgdiS</th>\n",
       "      <td>{'otherSubgenres': {'L': [{'S': 'rock'}, {'S':...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>006RpKEKItNO4q8TkAUpOv</th>\n",
       "      <td>{'otherSubgenres': {'L': [{'S': 'classical---c...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                   tracks\n",
       "000QWvZpHrBIVrW4dGbaVI  {'otherSubgenres': {'L': [{'S': 'electronic---...\n",
       "0010BnyFuw94XFautS2uJp  {'otherSubgenres': {'L': [{'S': ' world'}, {'S...\n",
       "0055LRFB7zfdCXDGodyIz3  {'otherSubgenres': {'L': []}, 'artist': {'S': ...\n",
       "005Dlt8Xaz3DkaXiRJgdiS  {'otherSubgenres': {'L': [{'S': 'rock'}, {'S':...\n",
       "006RpKEKItNO4q8TkAUpOv  {'otherSubgenres': {'L': [{'S': 'classical---c..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_path = join(data_path, \"labels.json\")\n",
    "labels = pd.read_json(label_path)\n",
    "labels.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5246a18b",
   "metadata": {},
   "source": [
    "As you can see, the dictionaries above contain a lot of useful information. Let's create a \n",
    "function to extract the data we want retrieve for our out recommendation system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c5f23315",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metadata(x):\n",
    "    cols = ['artist', 'genre', 'name', 'subgenres']\n",
    "    list_of_cols = []\n",
    "    for col in cols:\n",
    "        try:\n",
    "            mdata = list(x[col].values())[0]\n",
    "        except:\n",
    "            mdata = \"Unknown\"\n",
    "        list_of_cols.append(mdata)\n",
    "\n",
    "    return pd.Series(list_of_cols, index=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "584d3671",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>artist</th>\n",
       "      <th>genre</th>\n",
       "      <th>name</th>\n",
       "      <th>subgenres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000QWvZpHrBIVrW4dGbaVI</td>\n",
       "      <td>047</td>\n",
       "      <td>electronic</td>\n",
       "      <td>General Error</td>\n",
       "      <td>[{'S': 'electronic---synth-pop'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0010BnyFuw94XFautS2uJp</td>\n",
       "      <td>Jimmy Buffett</td>\n",
       "      <td>latin</td>\n",
       "      <td>La Vie Dansante</td>\n",
       "      <td>[{'S': 'latin---cubano'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0055LRFB7zfdCXDGodyIz3</td>\n",
       "      <td>New Order</td>\n",
       "      <td>rock</td>\n",
       "      <td>Doubts Even Here</td>\n",
       "      <td>[{'S': 'rock---new wave'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>005Dlt8Xaz3DkaXiRJgdiS</td>\n",
       "      <td>Ricardo Arjona</td>\n",
       "      <td>rock</td>\n",
       "      <td>Historia de Taxi</td>\n",
       "      <td>[{'S': 'rock---pop rock'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>006RpKEKItNO4q8TkAUpOv</td>\n",
       "      <td>Worrytrain</td>\n",
       "      <td>electronic</td>\n",
       "      <td>They Will Make My Passage Easy</td>\n",
       "      <td>[{'S': 'electronic---ambient'}]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    index          artist       genre   \n",
       "0  000QWvZpHrBIVrW4dGbaVI             047  electronic  \\\n",
       "1  0010BnyFuw94XFautS2uJp   Jimmy Buffett       latin   \n",
       "2  0055LRFB7zfdCXDGodyIz3       New Order        rock   \n",
       "3  005Dlt8Xaz3DkaXiRJgdiS  Ricardo Arjona        rock   \n",
       "4  006RpKEKItNO4q8TkAUpOv      Worrytrain  electronic   \n",
       "\n",
       "                             name                          subgenres  \n",
       "0                   General Error  [{'S': 'electronic---synth-pop'}]  \n",
       "1                 La Vie Dansante          [{'S': 'latin---cubano'}]  \n",
       "2                Doubts Even Here         [{'S': 'rock---new wave'}]  \n",
       "3                Historia de Taxi         [{'S': 'rock---pop rock'}]  \n",
       "4  They Will Make My Passage Easy    [{'S': 'electronic---ambient'}]  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_labels = labels['tracks'].apply(get_metadata).reset_index()\n",
    "clean_labels.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f44bb56d",
   "metadata": {},
   "source": [
    "The last piece of the puzzle is to clean the subgenres a bit, and to extract the path to each of the \n",
    "files since we will need them to load the recommendations in our app later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e602b999",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [electronic---synth-pop]\n",
       "1            [latin---cubano]\n",
       "2           [rock---new wave]\n",
       "3           [rock---pop rock]\n",
       "4      [electronic---ambient]\n",
       "Name: subgenres, dtype: object"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_vals(genres):\n",
    "    genre_list = []\n",
    "    for dicts in genres:\n",
    "        if type(dicts) != str:\n",
    "            for _, val in dicts.items():\n",
    "                genre_list.append(val)\n",
    "    return genre_list\n",
    "\n",
    "clean_labels['subgenres'] = clean_labels.subgenres.apply(get_vals)\n",
    "clean_labels['subgenres'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "eddd90d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2PaETSKl3w3IdtLIbDnQXJ</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/2PaETSKl3w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3Cu37dl54yhg2ZPrEnTx0O</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/3Cu37dl54y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4RTRzqkcvvkvuMK5IpFLmS</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/4RTRzqkcvv...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5A32KQZznC2HSqr9qzTl2N</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/5A32KQZznC...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2uPQvR5WBOI22Wj2gwwiT5</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/2uPQvR5WBO...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      ids                                               urls\n",
       "0  2PaETSKl3w3IdtLIbDnQXJ  ../data/ludwig_music_data/mp3/latin/2PaETSKl3w...\n",
       "1  3Cu37dl54yhg2ZPrEnTx0O  ../data/ludwig_music_data/mp3/latin/3Cu37dl54y...\n",
       "2  4RTRzqkcvvkvuMK5IpFLmS  ../data/ludwig_music_data/mp3/latin/4RTRzqkcvv...\n",
       "3  5A32KQZznC2HSqr9qzTl2N  ../data/ludwig_music_data/mp3/latin/5A32KQZznC...\n",
       "4  2uPQvR5WBOI22Wj2gwwiT5  ../data/ludwig_music_data/mp3/latin/2uPQvR5WBO..."
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_path = join(data_path, \"mp3\", \"latin\", \"*.mp3\")\n",
    "files = glob(file_path)\n",
    "ids = [i.split('/')[-1].replace(\".mp3\", '') for i in files]\n",
    "music_paths = pd.DataFrame(zip(ids, files), columns=[\"ids\", 'urls'])\n",
    "music_paths.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c00aeace",
   "metadata": {},
   "source": [
    "We'll combine all files with metadata into one dataframe and then format it as a list of JSON objects for our payload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "13039f12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ids</th>\n",
       "      <th>artist</th>\n",
       "      <th>genre</th>\n",
       "      <th>name</th>\n",
       "      <th>subgenres</th>\n",
       "      <th>urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0010BnyFuw94XFautS2uJp</td>\n",
       "      <td>Jimmy Buffett</td>\n",
       "      <td>latin</td>\n",
       "      <td>La Vie Dansante</td>\n",
       "      <td>[latin---cubano]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/0010BnyFuw...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>00RhgYVH6DrHl0SuZWDp8W</td>\n",
       "      <td>Jimmy Buffett</td>\n",
       "      <td>latin</td>\n",
       "      <td>Brown Eyed Girl</td>\n",
       "      <td>[latin---cubano]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/00RhgYVH6D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>01k69xxIQGL94F8IfIkI5l</td>\n",
       "      <td>Los Delinqüentes</td>\n",
       "      <td>latin</td>\n",
       "      <td>Fumata Del Ladrillo</td>\n",
       "      <td>[latin---flamenco, rock---punk]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/01k69xxIQG...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>02GUIyXZ9RNusgUocEQIzN</td>\n",
       "      <td>La Bottine Souriante</td>\n",
       "      <td>latin</td>\n",
       "      <td>Ma Paillasse</td>\n",
       "      <td>[latin---salsa]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/02GUIyXZ9R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>02IFfsWwxek6h9qLEH4sRA</td>\n",
       "      <td>Gipsy Kings</td>\n",
       "      <td>latin</td>\n",
       "      <td>Estrellas</td>\n",
       "      <td>[latin---flamenco]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/02IFfsWwxe...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                     ids                artist  genre   \n",
       "0      0  0010BnyFuw94XFautS2uJp         Jimmy Buffett  latin  \\\n",
       "1      1  00RhgYVH6DrHl0SuZWDp8W         Jimmy Buffett  latin   \n",
       "2      2  01k69xxIQGL94F8IfIkI5l      Los Delinqüentes  latin   \n",
       "3      3  02GUIyXZ9RNusgUocEQIzN  La Bottine Souriante  latin   \n",
       "4      4  02IFfsWwxek6h9qLEH4sRA           Gipsy Kings  latin   \n",
       "\n",
       "                  name                        subgenres   \n",
       "0      La Vie Dansante                 [latin---cubano]  \\\n",
       "1      Brown Eyed Girl                 [latin---cubano]   \n",
       "2  Fumata Del Ladrillo  [latin---flamenco, rock---punk]   \n",
       "3         Ma Paillasse                  [latin---salsa]   \n",
       "4            Estrellas               [latin---flamenco]   \n",
       "\n",
       "                                                urls  \n",
       "0  ../data/ludwig_music_data/mp3/latin/0010BnyFuw...  \n",
       "1  ../data/ludwig_music_data/mp3/latin/00RhgYVH6D...  \n",
       "2  ../data/ludwig_music_data/mp3/latin/01k69xxIQG...  \n",
       "3  ../data/ludwig_music_data/mp3/latin/02GUIyXZ9R...  \n",
       "4  ../data/ludwig_music_data/mp3/latin/02IFfsWwxe...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata = (music_data.select_columns(['index', 'ids'])\n",
    "                     .to_pandas()\n",
    "                     .merge(right=clean_labels, how=\"left\", left_on='ids', right_on='index')\n",
    "                     .merge(right=music_paths, how=\"left\", left_on='ids', right_on='ids')\n",
    "                     .drop(\"index_y\", axis=1)\n",
    "                     .rename({\"index_x\": \"index\"}, axis=1)\n",
    "        )\n",
    "metadata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "75f90784",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'artist': 'Jimmy Buffett',\n",
       "  'genre': 'latin',\n",
       "  'name': 'La Vie Dansante',\n",
       "  'subgenres': ['latin---cubano'],\n",
       "  'urls': '../data/ludwig_music_data/mp3/latin/0010BnyFuw94XFautS2uJp.mp3'},\n",
       " {'artist': 'Jimmy Buffett',\n",
       "  'genre': 'latin',\n",
       "  'name': 'Brown Eyed Girl',\n",
       "  'subgenres': ['latin---cubano'],\n",
       "  'urls': '../data/ludwig_music_data/mp3/latin/00RhgYVH6DrHl0SuZWDp8W.mp3'},\n",
       " {'artist': 'Los Delinqüentes',\n",
       "  'genre': 'latin',\n",
       "  'name': 'Fumata Del Ladrillo',\n",
       "  'subgenres': ['latin---flamenco', 'rock---punk'],\n",
       "  'urls': '../data/ludwig_music_data/mp3/latin/01k69xxIQGL94F8IfIkI5l.mp3'}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "payload = metadata.drop(['index', 'ids'], axis=1).to_dict(orient=\"records\")\n",
    "payload[:3]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "41785896",
   "metadata": {},
   "source": [
    "## 4. Embeddings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6b9e02a1",
   "metadata": {},
   "source": [
    "Audio embeddings are low dimensional vector representations of audio signals and they capture \n",
    "important features such as the pitch, timbre, and spatial characteristics of sound. These \n",
    "embeddings can be used as compact and meaningful representations of audio signals for various \n",
    "downstream audio processing tasks such as speech recognition, speaker recognition, music genre \n",
    "classification, and event detection. These embeddings are generally obtained using deep neural \n",
    "networks that take in an audio signal as input, and output a learned low-dimensional feature \n",
    "representation for that audio. In addition, these embeddings can also be used as input to \n",
    "further machine learning models.\n",
    "\n",
    "There are different ways in which we can get started creating embeddings for our songs:\n",
    "1. by training a deep neural network from scratch on our dataset and extracting the embedding layer, \n",
    "2. by using a pre-trained model and the transformers Python library, or \n",
    "3. by using purpose-built libraries like openl3 and pann_inference.\n",
    "\n",
    "There are other ways, of course, but here we'll use 2 and 3, the transformers architecture, and \n",
    "openl3 and pann_inference libraries.\n",
    "\n",
    "**Important INFO**: While there are three approached showcased here, you only need to pick one to \n",
    "continue with the tutorial. Here, we will follow along using the output from `panns_inference`.\n",
    "\n",
    "Let's get started."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2177117b",
   "metadata": {},
   "source": [
    "### openl3"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "33f37bcf",
   "metadata": {},
   "source": [
    "[OpenL3](https://github.com/marl/openl3/tree/main) is an open-source Python library for computing \n",
    "deep audio and image embeddings. It was created to provide an easy-to-use framework for extracting \n",
    "embeddings from audio and image data using pre-trained deep neural network models. The library \n",
    "includes pre-trained audio models like VGGish, YAMNet, and SoundNet, as well as pre-trained image \n",
    "models like ResNet and Inception. These models can be used for a variety of audio and image \n",
    "processing tasks, such as speech recognition, music genre classification, and object detection. Overall, \n",
    "OpenL3 is designed to make it easier for researchers and developers to incorporate deep learning \n",
    "models into their audio and image processing workflows.\n",
    "\n",
    "Let's read in an audio file and extract the embedding layer with openl3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6ed0d3c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1322496,)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_song = join(data_path, \"mp3\", \"latin\", \"0rXvhxGisD2djBmNkrv5Gt.mp3\")\n",
    "audio, sr = librosa.core.load(one_song, sr=44100, mono=True)\n",
    "audio.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e9ca22e",
   "metadata": {},
   "outputs": [],
   "source": [
    "player(audio, rate=sr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260a5f7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "open_emb, ts = openl3.get_audio_embedding(audio, sr, input_repr=\"mel128\", frontend='librosa')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ec388e21",
   "metadata": {},
   "source": [
    "The model returns an embedding vector for each timestamp and a timestamp vector. This means that \n",
    "to get a one dimensional embedding for the whole song, we'll need to get the mean of this vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48060114",
   "metadata": {},
   "outputs": [],
   "source": [
    "open_emb.shape, open_emb.mean(axis=0).shape, open_emb.mean(axis=0)[:20]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "29452242",
   "metadata": {},
   "source": [
    "You can generate your embedding layer for the whole dataset with the following function. Note that \n",
    "loading the model first, in particular Kapre, will work on a GPU without any further configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6308f9c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_kapre = openl3.models.load_audio_embedding_model(\n",
    "    input_repr='mel128', content_type='music', embedding_size=512\n",
    ")\n",
    "\n",
    "def get_open_embs(batch):\n",
    "    audio_arrays = [song['array'] for song in batch['audio']]\n",
    "    sr_arrays = [song['sampling_rate'] for song in batch['audio']]\n",
    "    embs_list, _ = openl3.get_audio_embedding(audio_arrays, sr_arrays, model=model_kapre)\n",
    "    batch[\"open_embeddings\"] = np.array([embedding.mean(axis=0) for embedding in embs_list])\n",
    "    return batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f072ad36",
   "metadata": {},
   "outputs": [],
   "source": [
    "music_data = music_data.map(get_open_embs, batched=True, batch_size=20)\n",
    "music_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "429933db",
   "metadata": {},
   "source": [
    "The nice thing about openl3 is that it comes with the best model for our task. The downside is \n",
    "that it is the slowest of the three methods showcased here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "17ea9e90",
   "metadata": {},
   "source": [
    "### Panns Inference"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "23f6e094",
   "metadata": {},
   "source": [
    "The `panns_inference` library is a Python package built on top of PyTorch and torchaudio that \n",
    "provides an interface for audio tagging and sound event detection tasks. It implements CNN-based \n",
    "models trained on large-scale audio datasets such as AudioSet and UrbanSound8K. The package was \n",
    "created to make it easy for researchers and practitioners to use these pre-trained models for \n",
    "inference on their own audio datasets, without needing to train their own models from scratch. The \n",
    "`panns_inference` library provides a high-level, user-friendly API for loading pre-trained models, \n",
    "generating embeddings, and performing audio classification tasks in just a few lines of code.\n",
    "\n",
    "The `panns_inference` package requires that the data is either as a numpy array or as a torch \n",
    "tensor, both of shape `[batch, vector]` so let's reshape our song."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "371b47bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 1322496)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "audio2 = audio[None, :]\n",
    "audio2.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0453c3b5",
   "metadata": {},
   "source": [
    "Bare in mind that this next step, downloading the model, can take quite a bit of time depending on \n",
    "your internet speed. Afterwards, inference is quite fast and the model will return to us two \n",
    "vectors, the timestamps and the embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3c5531bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checkpoint path: /home/ramonperez/panns_data/Cnn14_mAP=0.431.pth\n",
      "GPU number: 1\n"
     ]
    }
   ],
   "source": [
    "at = AudioTagging(checkpoint_path=None, device='cuda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fd2d6dcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "clipwise_output, embedding = at.inference(audio2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "98e724c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 527), (1, 2048))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clipwise_output.shape, embedding.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "35bb1792",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.       , 0.       , 0.       , 0.       , 0.       , 0.       ,\n",
       "       3.1233616, 0.       , 0.       , 0.       , 0.       , 0.       ,\n",
       "       0.       , 0.       , 0.       , 0.       , 0.       , 0.       ,\n",
       "       0.       , 1.6375436, 0.       , 0.       , 0.       , 0.       ,\n",
       "       0.       , 0.       , 0.       , 0.       , 0.       , 0.       ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding[0, 470:500]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2fee11eb",
   "metadata": {},
   "source": [
    "To get an embedding layer for all of the songs using the `panns_inference` package, you can use the following \n",
    "function. This is the output we will be using for the remainder of the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7fdb2fb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_panns_embs(batch):\n",
    "    arrays = [torch.tensor(val['array'], dtype=torch.float64) for val in batch['audio']]\n",
    "    inputs = torch.nn.utils.rnn.pad_sequence(arrays, batch_first=True, padding_value=0).type(torch.cuda.FloatTensor)\n",
    "    _, embedding = at.inference(inputs)\n",
    "    batch['panns_embeddings'] = embedding\n",
    "    return batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ea8c573f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2501080e5bcd4c38b9adad279de98d21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/979 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['audio', 'index', 'ids', 'panns_embeddings'],\n",
       "    num_rows: 979\n",
       "})"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_data = music_data.map(get_panns_embs, batched=True, batch_size=8)\n",
    "music_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "913b6407",
   "metadata": {},
   "source": [
    "### Transformers"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1f1240fb",
   "metadata": {},
   "source": [
    "Transformers are a type of neural network used for natural language processing, but the architecture \n",
    "can also be used for processing audio data by breaking the sound waves into smaller parts and learning \n",
    "how those parts fit together to form meaning.\n",
    "\n",
    "We can load a pre-trained model from the Hugging Face hub and extract the embeddings from it. Note \n",
    "that this step will give us the worst result of the three since Wav2Vec was trained to recognize \n",
    "speech rather than to classify music genres. Hence, it is important to note that fine-tunning the \n",
    "data with Wav2Vec might not improve a whole lot the quality of the embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "124b9c51",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ramonperez/anaconda/envs/qdrant101/lib/python3.10/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.\n",
      "  warnings.warn(\n",
      "Some weights of the model checkpoint at facebook/wav2vec2-base were not used when initializing Wav2Vec2Model: ['quantizer.weight_proj.bias', 'quantizer.codevectors', 'project_hid.bias', 'project_q.weight', 'project_q.bias', 'quantizer.weight_proj.weight', 'project_hid.weight']\n",
      "- This IS expected if you are initializing Wav2Vec2Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing Wav2Vec2Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = AutoModel.from_pretrained('facebook/wav2vec2-base').to(device)\n",
    "feature_extractor = AutoFeatureExtractor.from_pretrained('facebook/wav2vec2-base')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "66c020b2",
   "metadata": {},
   "source": [
    "A key step before extracting the features from each song and passing them through the model \n",
    "is to resample the songs 16kHz."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "537625a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "resampled_audio = librosa.resample(y=audio2, orig_sr=sr, target_sr=16_000)\n",
    "display(player(resampled_audio, rate=16_000))\n",
    "resampled_audio.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e4729d61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 16000])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = feature_extractor(\n",
    "    resampled_audio[0], sampling_rate=feature_extractor.sampling_rate, return_tensors=\"pt\",\n",
    "    padding=True, return_attention_mask=True, truncation=True, max_length=16_000\n",
    ").to(device)\n",
    "\n",
    "inputs['input_values'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 768])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    embeddings = model(**inputs).last_hidden_state.mean(dim=1)\n",
    "embeddings.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "43808bfd",
   "metadata": {},
   "source": [
    "To generate the embedding layer for the whole dataset, we can use the following function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c726515b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_trans_embs(batch):\n",
    "    audio_arrays = [x[\"array\"] for x in batch[\"audio\"]]\n",
    "\n",
    "    inputs = feature_extractor(\n",
    "        audio_arrays, sampling_rate=16_000, return_tensors=\"pt\", padding=True, \n",
    "        return_attention_mask=True, max_length=16_000, truncation=True\n",
    "    ).to(device)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        pooled_embeds = model(**inputs).last_hidden_state.mean(dim=1)\n",
    "    \n",
    "    return {\"transform_embeddings\": pooled_embeds.cpu().numpy()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84d65979",
   "metadata": {},
   "outputs": [],
   "source": [
    "music_data = music_data.cast_column(\"audio\", Audio(sampling_rate=16_000))\n",
    "music_data = music_data.map(embed_audio, batched=True, batch_size=20)\n",
    "music_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42ace83a",
   "metadata": {},
   "source": [
    "## 5. Building a Recommendation System"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e89b130d",
   "metadata": {},
   "source": [
    "Recommendation systems are algorithms and techniques used to suggest items or content to users \n",
    "based on their preferences, historical data, or behavior. These systems aim to provide personalized \n",
    "recommendations to users, helping them discover new items of interest and enhancing their overall \n",
    "user experience. Recommendation systems are widely used in various domains such as e-commerce, \n",
    "streaming platforms, social media, and more.\n",
    "\n",
    "Let's start by populating the collection we created earlier. If you picked the transformers approach \n",
    "or openl3 to follow along, you will need to recreate your collection with the appropriate dimension size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f729ef1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UpdateResult(operation_id=0, status=<UpdateStatus.COMPLETED: 'completed'>)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.upsert(\n",
    "    collection_name=my_collection,\n",
    "    points=models.Batch(\n",
    "        ids=music_data['index'],\n",
    "        vectors=music_data['panns_embeddings'],\n",
    "        payloads=payload\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6ee0d96a",
   "metadata": {},
   "source": [
    "We can retrieve any song by its id using `client.retrieve()` and then extract the information \n",
    "in the payload with the `.payload` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "590e82b7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = client.retrieve(\n",
    "    collection_name=my_collection,\n",
    "    ids=[100],\n",
    "    with_vectors=True # we can turn this on and off depending on our needs\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "8a5044c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'artist': 'La Bottine Souriante',\n",
       " 'genre': 'latin',\n",
       " 'name': 'Chant de la luette',\n",
       " 'subgenres': ['latin---salsa'],\n",
       " 'urls': '../data/ludwig_music_data/mp3/latin/0lyeChzw7IWf9ytZ7S0jDK.mp3'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result[0].payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2c3d035",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = librosa.core.load(result[0].payload['urls'], sr=44100, mono=True)\n",
    "player(r[0], rate=r[1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6c4f512b",
   "metadata": {},
   "source": [
    "<iframe style=\"border-radius:12px\" src=\"https://open.spotify.com/embed/track/0lyeChzw7IWf9ytZ7S0jDK?utm_source=generator\" width=\"100%\" height=\"152\" frameBorder=\"0\" allowfullscreen=\"\" allow=\"autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture\" loading=\"lazy\"></iframe>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8ba504ba",
   "metadata": {},
   "source": [
    "You can search for similar songs with the `client.search()` method. Let's find and artist and a song \n",
    "we like and use that id to grab the embedding and search for similar songs.\n",
    "\n",
    "PS. Here is [Celia Cruz](https://www.youtube.com/watch?v=AXN-_asIaYs&ab_channel=GravityLimited). 😎"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3d3b2c07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ids</th>\n",
       "      <th>artist</th>\n",
       "      <th>genre</th>\n",
       "      <th>name</th>\n",
       "      <th>subgenres</th>\n",
       "      <th>urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>122</td>\n",
       "      <td>0v1oaOqkXpubdykx58BQwY</td>\n",
       "      <td>Celia Cruz</td>\n",
       "      <td>latin</td>\n",
       "      <td>Juancito Trucupey</td>\n",
       "      <td>[latin---salsa]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/0v1oaOqkXp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>150</td>\n",
       "      <td>19zWrDlXew0Fzouu7a4qhx</td>\n",
       "      <td>Celia Cruz</td>\n",
       "      <td>latin</td>\n",
       "      <td>Cuando Sali De Cuba</td>\n",
       "      <td>[latin---salsa]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/19zWrDlXew...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>178</td>\n",
       "      <td>1MYds6o9aN2Wxa4TDxcJPB</td>\n",
       "      <td>Celia Cruz</td>\n",
       "      <td>latin</td>\n",
       "      <td>Mi vida es cantar</td>\n",
       "      <td>[latin---salsa]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/1MYds6o9aN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>459</td>\n",
       "      <td>3WphzI2fb2NTUsfja51U7P</td>\n",
       "      <td>Celia Cruz</td>\n",
       "      <td>latin</td>\n",
       "      <td>Dile que por mi no tema</td>\n",
       "      <td>[latin---salsa]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/3WphzI2fb2...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     index                     ids      artist  genre   \n",
       "122    122  0v1oaOqkXpubdykx58BQwY  Celia Cruz  latin  \\\n",
       "150    150  19zWrDlXew0Fzouu7a4qhx  Celia Cruz  latin   \n",
       "178    178  1MYds6o9aN2Wxa4TDxcJPB  Celia Cruz  latin   \n",
       "459    459  3WphzI2fb2NTUsfja51U7P  Celia Cruz  latin   \n",
       "\n",
       "                        name        subgenres   \n",
       "122        Juancito Trucupey  [latin---salsa]  \\\n",
       "150      Cuando Sali De Cuba  [latin---salsa]   \n",
       "178        Mi vida es cantar  [latin---salsa]   \n",
       "459  Dile que por mi no tema  [latin---salsa]   \n",
       "\n",
       "                                                  urls  \n",
       "122  ../data/ludwig_music_data/mp3/latin/0v1oaOqkXp...  \n",
       "150  ../data/ludwig_music_data/mp3/latin/19zWrDlXew...  \n",
       "178  ../data/ludwig_music_data/mp3/latin/1MYds6o9aN...  \n",
       "459  ../data/ludwig_music_data/mp3/latin/3WphzI2fb2...  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.query(\"artist == 'Celia Cruz'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "2eada85f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredPoint(id=150, version=0, score=0.99999994, payload={'artist': 'Celia Cruz', 'genre': 'latin', 'name': 'Cuando Sali De Cuba', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/19zWrDlXew0Fzouu7a4qhx.mp3'}, vector=None),\n",
       " ScoredPoint(id=730, version=0, score=0.9206133, payload={'artist': 'Cartola', 'genre': 'latin', 'name': 'Fita meus olhos', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/5iyRJ796USPTXEO4JXO0gC.mp3'}, vector=None),\n",
       " ScoredPoint(id=251, version=0, score=0.9087784, payload={'artist': \"Oscar D'León\", 'genre': 'latin', 'name': 'Volver a Verte', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/1kD5EOoZ45kjq50NLfhRGc.mp3'}, vector=None),\n",
       " ScoredPoint(id=739, version=0, score=0.90295744, payload={'artist': 'Cartola', 'genre': 'latin', 'name': 'Verde que te quero rosa', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/5plwAx4oAWnuhSwivS5Yeg.mp3'}, vector=None),\n",
       " ScoredPoint(id=268, version=0, score=0.8995003, payload={'artist': 'Chicha Libre', 'genre': 'latin', 'name': 'La cumbia del zapatero', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/1ufmU58QldvKrHuATBb3kU.mp3'}, vector=None),\n",
       " ScoredPoint(id=766, version=0, score=0.88916755, payload={'artist': 'Ska Cubano', 'genre': 'latin', 'name': 'Tequila', 'subgenres': ['latin---cubano', 'reggae'], 'urls': '../data/ludwig_music_data/mp3/latin/618iBzv4oH2wb0WElQV9ru.mp3'}, vector=None),\n",
       " ScoredPoint(id=7, version=0, score=0.8882055, payload={'artist': 'Ibrahim Ferrer', 'genre': 'latin', 'name': 'Nuestra Ruca', 'subgenres': ['latin---cubano'], 'urls': '../data/ludwig_music_data/mp3/latin/02vPUwCweGxigItnNf2Jfr.mp3'}, vector=None),\n",
       " ScoredPoint(id=467, version=0, score=0.88348734, payload={'artist': 'La-33', 'genre': 'latin', 'name': 'Soledad', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/3bpqoOSDwdaK003DPMvDJQ.mp3'}, vector=None),\n",
       " ScoredPoint(id=388, version=0, score=0.882995, payload={'artist': 'David Byrne', 'genre': 'latin', 'name': 'Loco De Amor', 'subgenres': ['latin---salsa', 'latin---samba', 'rock---pop rock'], 'urls': '../data/ludwig_music_data/mp3/latin/2uJsn2yi8HVZ8qwICHcNSW.mp3'}, vector=None),\n",
       " ScoredPoint(id=139, version=0, score=0.8820398, payload={'artist': 'Ibrahim Ferrer', 'genre': 'latin', 'name': 'Qué bueno baila usted', 'subgenres': ['latin---cubano'], 'urls': '../data/ludwig_music_data/mp3/latin/16FEEqvnZKcgfA5esxe5kL.mp3'}, vector=None)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.search(\n",
    "    collection_name=my_collection,\n",
    "    query_vector=music_data[150]['panns_embeddings'],\n",
    "    limit=10\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7304fede",
   "metadata": {},
   "source": [
    "You can evaluate the search results by looking at the score or by listening to the songs and \n",
    "judging how similar they really are. I, the author, can vouch for the quality of the ones we \n",
    "got for Celia Cruz. 😎\n",
    "\n",
    "The recommendation API works a bit differently, we don't need a vector query but rather the \n",
    "ids of positive (required) vectors and negative (optional) ones, and Qdrant will do the heavy \n",
    "lifting for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "42ddfcd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredPoint(id=384, version=0, score=0.96683824, payload={'artist': 'Gilberto Santa Rosa', 'genre': 'latin', 'name': 'Perdoname', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/2qqrgPaRZow7lrLttDL6Im.mp3'}, vector=None),\n",
       " ScoredPoint(id=424, version=0, score=0.9633477, payload={'artist': 'Gilberto Santa Rosa', 'genre': 'latin', 'name': 'Amanecer Borincano', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/39FQfusOwKnPCjOgQHcx6S.mp3'}, vector=None),\n",
       " ScoredPoint(id=190, version=0, score=0.9624174, payload={'artist': 'Luigi Texidor', 'genre': 'latin', 'name': 'Mi Testamento', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/1RIdI5c7RjjagAcMA5ixpv.mp3'}, vector=None),\n",
       " ScoredPoint(id=92, version=0, score=0.95979774, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Mambo Gozón', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/0hk1gSyn3wKgdxqF6qaKUZ.mp3'}, vector=None),\n",
       " ScoredPoint(id=886, version=0, score=0.95851713, payload={'artist': 'Tony Vega', 'genre': 'latin', 'name': 'Ella es', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/718X6sjlHdmOzdTfJv4tUc.mp3'}, vector=None)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.recommend(\n",
    "    collection_name=my_collection,\n",
    "    positive=[178, 122],\n",
    "    limit=5\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "32faf2a7",
   "metadata": {},
   "source": [
    "Say we don't like [Chayanne](https://www.youtube.com/watch?v=GQa10n21YIw&ab_channel=AnnunziattaGibson) \n",
    "because his songs are too mushy. We can use the id of one of his mushiest songs so that Qdrant gets \n",
    "us results as far away as possible from such a song."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c0bcb92d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>ids</th>\n",
       "      <th>artist</th>\n",
       "      <th>genre</th>\n",
       "      <th>name</th>\n",
       "      <th>subgenres</th>\n",
       "      <th>urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>162</td>\n",
       "      <td>1EyREvPFfh2TgXFMCPoydD</td>\n",
       "      <td>Chayanne</td>\n",
       "      <td>latin</td>\n",
       "      <td>Caprichosa</td>\n",
       "      <td>[latin---salsa, pop---ballad]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/1EyREvPFfh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>208</th>\n",
       "      <td>208</td>\n",
       "      <td>1XMw83NJw29iwarOqVibos</td>\n",
       "      <td>Chayanne</td>\n",
       "      <td>latin</td>\n",
       "      <td>Querida</td>\n",
       "      <td>[latin---samba, pop---ballad]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/1XMw83NJw2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>385</td>\n",
       "      <td>2sKo5u6IppUEudIz265wYa</td>\n",
       "      <td>Chayanne</td>\n",
       "      <td>latin</td>\n",
       "      <td>Yo Te Amo</td>\n",
       "      <td>[latin---salsa, pop---ballad]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/2sKo5u6Ipp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>412</th>\n",
       "      <td>412</td>\n",
       "      <td>34hM4PLlhyBysgL50IWdHf</td>\n",
       "      <td>Chayanne</td>\n",
       "      <td>latin</td>\n",
       "      <td>Y tú te vas</td>\n",
       "      <td>[latin---salsa, pop---ballad]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/34hM4PLlhy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>645</th>\n",
       "      <td>645</td>\n",
       "      <td>4zkOTmiamebLJ39Sqbp7sb</td>\n",
       "      <td>Chayanne</td>\n",
       "      <td>latin</td>\n",
       "      <td>Boom Boom</td>\n",
       "      <td>[latin---salsa, pop---ballad]</td>\n",
       "      <td>../data/ludwig_music_data/mp3/latin/4zkOTmiame...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     index                     ids    artist  genre         name   \n",
       "162    162  1EyREvPFfh2TgXFMCPoydD  Chayanne  latin   Caprichosa  \\\n",
       "208    208  1XMw83NJw29iwarOqVibos  Chayanne  latin      Querida   \n",
       "385    385  2sKo5u6IppUEudIz265wYa  Chayanne  latin    Yo Te Amo   \n",
       "412    412  34hM4PLlhyBysgL50IWdHf  Chayanne  latin  Y tú te vas   \n",
       "645    645  4zkOTmiamebLJ39Sqbp7sb  Chayanne  latin    Boom Boom   \n",
       "\n",
       "                         subgenres   \n",
       "162  [latin---salsa, pop---ballad]  \\\n",
       "208  [latin---samba, pop---ballad]   \n",
       "385  [latin---salsa, pop---ballad]   \n",
       "412  [latin---salsa, pop---ballad]   \n",
       "645  [latin---salsa, pop---ballad]   \n",
       "\n",
       "                                                  urls  \n",
       "162  ../data/ludwig_music_data/mp3/latin/1EyREvPFfh...  \n",
       "208  ../data/ludwig_music_data/mp3/latin/1XMw83NJw2...  \n",
       "385  ../data/ludwig_music_data/mp3/latin/2sKo5u6Ipp...  \n",
       "412  ../data/ludwig_music_data/mp3/latin/34hM4PLlhy...  \n",
       "645  ../data/ludwig_music_data/mp3/latin/4zkOTmiame...  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.query(\"artist == 'Chayanne'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5d159bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredPoint(id=546, version=0, score=0.87100524, payload={'artist': '¡Cubanismo!', 'genre': 'latin', 'name': 'El Preguntón', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/4EH5vM8p1Ibvlz5cgZLHvY.mp3'}, vector=None),\n",
       " ScoredPoint(id=85, version=0, score=0.86223793, payload={'artist': '¡Cubanismo!', 'genre': 'latin', 'name': 'Malembe', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/0efiEWiAFtHrQHTWfeDikg.mp3'}, vector=None),\n",
       " ScoredPoint(id=910, version=0, score=0.8605486, payload={'artist': '¡Cubanismo!', 'genre': 'latin', 'name': 'Cubanismo Llegó', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/7FSSdHxCoyEMfHUP6NdOb2.mp3'}, vector=None),\n",
       " ScoredPoint(id=540, version=0, score=0.85953826, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Cual Es La Idea', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/4CNCGwxNp9rnVqo2fzmDYK.mp3'}, vector=None),\n",
       " ScoredPoint(id=812, version=0, score=0.85860175, payload={'artist': 'Tommy Olivencia', 'genre': 'latin', 'name': 'Trucutú', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/6I9OiSVppRGjuAweyBucE2.mp3'}, vector=None)]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.recommend(\n",
    "    collection_name=my_collection,\n",
    "    positive=[178, 122],\n",
    "    negative=[385],\n",
    "    limit=5\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f9646e73",
   "metadata": {},
   "source": [
    "Say we want to get recommendations based on a song we just recently listened to and liked, \n",
    "and that the system remembers all of our preferences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "98f30cb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "marc_anthony_valio_la_pena = music_data[301]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e7afb782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredPoint(id=546, version=0, score=0.86705625, payload={'artist': '¡Cubanismo!', 'genre': 'latin', 'name': 'El Preguntón', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/4EH5vM8p1Ibvlz5cgZLHvY.mp3'}, vector=None),\n",
       " ScoredPoint(id=85, version=0, score=0.8635909, payload={'artist': '¡Cubanismo!', 'genre': 'latin', 'name': 'Malembe', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/0efiEWiAFtHrQHTWfeDikg.mp3'}, vector=None),\n",
       " ScoredPoint(id=540, version=0, score=0.8588973, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Cual Es La Idea', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/4CNCGwxNp9rnVqo2fzmDYK.mp3'}, vector=None),\n",
       " ScoredPoint(id=812, version=0, score=0.85626286, payload={'artist': 'Tommy Olivencia', 'genre': 'latin', 'name': 'Trucutú', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/6I9OiSVppRGjuAweyBucE2.mp3'}, vector=None),\n",
       " ScoredPoint(id=587, version=0, score=0.85231805, payload={'artist': 'Tito Puente & His Orchestra', 'genre': 'latin', 'name': 'Mambo Gozon', 'subgenres': ['latin---salsa'], 'urls': '../data/ludwig_music_data/mp3/latin/4Sewxyw6EtUldCIz2sD9S5.mp3'}, vector=None)]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.recommend(\n",
    "    collection_name=my_collection,\n",
    "    positive=[marc_anthony_valio_la_pena['idx'], 178, 122, 459],\n",
    "    negative=[385],\n",
    "    limit=5\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d0075385",
   "metadata": {},
   "source": [
    "Lastly, imagine we want a [Samba](https://www.youtube.com/watch?v=m3WtyqJzLAI&ab_channel=luka%C4%8Denko7) \n",
    "filter for the recommendations we get, the UI could have tags for us to choose from and Qdrant would do the rest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f51f9ade",
   "metadata": {},
   "outputs": [],
   "source": [
    "samba_songs = models.Filter(\n",
    "    must=[models.FieldCondition(key=\"subgenres\", match=models.MatchAny(any=['latin---samba']))]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "25ac6cb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ScoredPoint(id=540, version=0, score=0.8588973, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Cual Es La Idea', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/4CNCGwxNp9rnVqo2fzmDYK.mp3'}, vector=None),\n",
       " ScoredPoint(id=493, version=0, score=0.8236424, payload={'artist': 'Tito Nieves', 'genre': 'latin', 'name': 'De mi enamórate', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/3nnQUYKWBmHlfm5XpdWqNr.mp3'}, vector=None),\n",
       " ScoredPoint(id=92, version=0, score=0.8120091, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Mambo Gozón', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/0hk1gSyn3wKgdxqF6qaKUZ.mp3'}, vector=None),\n",
       " ScoredPoint(id=856, version=0, score=0.80171, payload={'artist': 'Tito Puente', 'genre': 'latin', 'name': 'Son de la Loma', 'subgenres': ['latin---samba'], 'urls': '../data/ludwig_music_data/mp3/latin/6c8qeNyZrTB8E3RKdPdNBh.mp3'}, vector=None),\n",
       " ScoredPoint(id=892, version=0, score=0.7895387, payload={'artist': 'David Byrne', 'genre': 'latin', 'name': 'Make Believe Mambo', 'subgenres': ['latin---salsa', 'latin---samba', 'rock---pop rock'], 'urls': '../data/ludwig_music_data/mp3/latin/74V0PhSWlBtHvBQAMYMgsX.mp3'}, vector=None)]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = client.recommend(\n",
    "    collection_name=my_collection,\n",
    "    query_filter=samba_songs,\n",
    "    positive=[marc_anthony_valio_la_pena['idx'], 178, 122, 459],\n",
    "    negative=[385],\n",
    "    limit=5\n",
    ")\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "181031a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for result in results:\n",
    "    song, sr = librosa.core.load(result.payload['urls'], sr=44100, mono=True)\n",
    "    display(player(song, rate=sr))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "be1f3af8",
   "metadata": {},
   "source": [
    "That's it! So, what's next? You should try using different genres (or all of them), creating embeddings \n",
    "for these and then building your own recommendation engine on top of Qdrant. Better yet, you could \n",
    "find your own dataset and build a personalized search engine for the things you like, just make sure \n",
    "you let us know via our [discord channel here](https://qdrant.to/discord). 😎"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b8904f6c",
   "metadata": {},
   "source": [
    "## 6. Putting it All Together"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "baece8b0",
   "metadata": {},
   "source": [
    "Now that we have covered everything we need, it is time to put it to the test with a UI, and \n",
    "for this, we'll use [streamlit](https://streamlit.io/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a4b3745-1011-4a6a-b9a2-aae128c4ea88",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile recsys_app.py\n",
    "\n",
    "from panns_inference import AudioTagging\n",
    "from qdrant_client import QdrantClient\n",
    "from pedalboard.io import AudioFile\n",
    "import streamlit as st\n",
    "import torch\n",
    "\n",
    "st.title(\"Music Recommendation App\")\n",
    "st.markdown(\"Upload your favorite songs and get a list of recommendations from our database of music.\")\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "at = AudioTagging(checkpoint_path=None, device=device)\n",
    "client = QdrantClient(\"localhost\", port=6333)\n",
    "\n",
    "music_file = st.file_uploader(label=\"📀 Music file 🎸\",)\n",
    "\n",
    "if music_file:\n",
    "    st.audio(music_file)\n",
    "\n",
    "    with AudioFile(music_file) as f:\n",
    "        a_song = f.read(f.frames)[0][None, :]\n",
    "\n",
    "    clip, emb = at.inference(a_song)\n",
    "\n",
    "    st.markdown(\"## Semantic Search\")\n",
    "    results = client.search(collection_name=\"music_collection\", query_vector=emb[0], limit=4)\n",
    "    \n",
    "    for result in results:\n",
    "        st.header(f\"Song: {result.payload['name']}\")\n",
    "        st.subheader(f\"Artist: {result.payload['artist']}\")\n",
    "        st.audio(result.payload[\"urls\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13b90973",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!streamlit run recsys_app.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
